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Abstract: The topic of assistive technology 
(AT) outcomes has only recently received 
attention in the professional literature. As a 
result, there is a considerable void in the 
profession’s ability to address contemporary 
questions about the value and use of AT. The 
purpose of this article is to highlight the 
theory, development, and research efforts of 
the ATOMS Project as it seeks to create a 
prototype of an AT outcome measurement 
system. Specific attention is devoted to 
research efforts to socially validate selected 
components of a proposed outcome system. 
The results reveal significant support for a 
system that utilizes a theoretical framework; 
involves paper or electronic format data 
collection instmments that do not require 
extensive training and expertise; assimilates 
data from multiple sources; and provides 
visual representation of the data to facilitate 
interpretation and decision-making. The 
benefits and outcomes of this research and 
development agenda are described. 
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Fuhrer (1999) observed that interest in the 
outcomes of assistive technology (AT) is a 
relatively recent phenomena. Support for this 
observation is easily gathered by reviewing the 
journal literature and leading personnel 
preparation textbooks. Prior to 1996, there is 
little evidence to indicate that the profession 
was concerned about issues associated with 
the collection and use of AT outcome data. 
Apparently, we never asked ourselves for 
evidence concerning the impact of AT. It was 


obvious that AT was valuable for an individual 
with a disability. We observed a problem, 
provided appropriate AT devices and services, 
and then watched the transformation that 
occurred when an individual completed a task 
that was formerly difficult or impossible to 
do. To the extent that we sought to collect 
data, we simply asked the individual if they 
liked the new device and whether they found 
it helpful. In hindsight, we appear so naive. 

An Emerging Field 

Arguably, several milestones can be 
documented that served to increase the 
profession’s awareness and sensitivity about 
the need to begin asking questions regarding 
the measurement of AT outcomes (The 
ATOMS Project, 2003). Early works raised 
questions about whether or not the profession 
would make the commitment to measuring 
assistive technology outcomes (DeRuyter, 
1995) and why outcome data was essential for 
addressing questions about the quality of 
service delivery systems (DeRuyter, 1997). 
The first special issue of a journal devoted to 
AT outcomes appeared less than a decade ago 
(Smith, 1996) with a second special issue 
following four years later (Edyburn, 2000). 
Thus, the first indications of an emerging 
discipline focusing on measuring AT 
outcomes can be found in the journal 
literature. 

Developmentally, the discipline of AT 
outcome measurement is less than 10 years 
old. The emerging literature can be 
characterized as philosophical and theoretical 
as leaders clarify the importance of the 
research and development agenda. As a result, 
there is an urgent need for maturation of 


Assistive Technology Outcomes and Benefits / 8 



Fall 2004, Vol. 1, Num. 1 


measurement theory and instrumentation 
development. 

RESNA (1998a, 1998b, 1998c) published a 
three-volume monograph with the results of a 
state-of-the-art survey of AT outcome 
assessment practices. The findings revealed 
that a majority of the instmments used by 
practitioners for measuring the outcomes of 
AT were self-developed with unknown 
technical adequacy qualities. This landmark 
work graphically illustrated the dismal 
condition the profession was in relative to AT 
outcome measurement. 

The current state of AT outcome 
measurement can also be understood from 
the results of two studies that have sought to 
extract AT outcome data from large extant 
data sets. While the findings provide a glimpse 
of the number of individuals that use AT, they 
are also disappointing as we have learned that 
there are serious flaws in current professional 
practice such that outcome data are not 
routinely collected (Carlson, Ehrlich, Berland, 
& Bailey, 2001; Moser, 2003). 

Increased awareness about the deficits in the 
AT outcome knowledge base and the dawn of 
the 21st century created a context of increased 
accountability and desire for understanding 
the value of technology investments. 
Recognition of these issues resulted in the 
establishment of three national research 
centers to advance an agenda to substantially 
increasing the knowledge base surrounding 
AT and its effective use by individuals with 
disabilities. 

The Office of Special Education Programs 
(OSEP) funded the National Assistive 
Technology Research Institute (NATRI) 
based at the University of Kentucky. This 
center is charged with conducting assistive 
research, translating research into assistive 
technology practice, and providing resources 
to improve the delivery of AT services. 


Several in-progress studies hold considerable 
potential for informing state and federal 
policy concerning effective AT practices 
(Lahm, Bausch, Hasselbring, & Blackhurst, 
2001). To learn more about this center, visit 
the NATRI home page: http://natri.uky.edu . 

A second federal agency was also concerned 
about AT and has funded priorities to 
advance a research agenda concerning 
assistive technology outcomes. In October 
2001, National Institute on Disability and 
Rehabilitation Research (NIDRR) funded 
two, five-year, research centers to address the 
gap in data collection efforts concerning AT 
outcomes, as well as the paucity of 
measurement instruments and strategies. The 
Assistive Technology Outcomes 

Measurement System (ATOMS) Project is 
based at the University of Wisconsin- 
Milwaukee. To learn more about this center, 
visit: http:/ / www.atoms.uwm.edu . The 

Consortium for Assistive Technology 

Outcome Research (CATOR) is housed at 
Duke University. To learn more about this 
center, visit: http://www.atoutcomes.org . 

Given the lack of data on AT outcomes and 
the importance of such information for a wide 
variety of stakeholders (i.e., individuals with 
disabilities, AT service providers, 

administrators, funding agencies, AT 
developers), one of the key activities of the 
ATOMS Project has focused on the 
development of a prototype of a large-scale 
AT outcome measurement system. The 
purpose of this article is to describe the theory 
development underlying such a system and 
preliminary research that has been conducted 
to socially validate the components. 

Method 

In order to begin operationalizing a vision of 
what a future AT outcome system might look 
like, the ATOMS Project has engaged in a 
number of research and development 


Assistive Technology Outcomes and Benefits / 9 





Fall 2004, Vol. 1, Num. 1 


activities to design a prototype. These 
activities include conducting an extensive 
number of field scans to ascertain the what 
efforts have been devoted to measuring the 
outcomes of AT and determining what types 
of innovative research methodologies might 
be suitable for collecting reliable and valid 
outcome data to inform AT decision-making. 

For the purpose of this investigation, seven 
components of a proposed AT outcome 
system were identified by the research team 
for social validation. The seven components 
included: (a) theoretical framework; (b) 
expertise, training, and availability of the 
assessment instruments; (c) data collection 
techniques; (d) data assimilation; (e) data 
reduction and visualization; (f) dynamic 
norming; and (g) data-based decision-making. 
Each component is described briefly below. 

Components of an AT Outcome System 

Previous research by the ATOMS Project 
suggested that the constmct of AT outcome 
may be multidimensional (change in 
performance/function, change in 

participation, usage (why or why not), 
consumer satisfaction (process, devices), goal 
achievement, quality of life, and cost) rather 
than something that can be captured in a 
single score (Edyburn, 2003). In addition, 
significant methodological challenges remain 
to be resolved on how to isolate and discern 
the specific impact of AT as it is frequently 
implemented concurrently with other 
interventions (Smith, 2002). As a result, 
ATOMS Project researchers believe there is a 
significant need for building AT outcomes 
systems that are grounded in a theoretical 
framework. 

Test developers use a continuum of 
approaches for designing assessment 
instruments: from informal assessment tools 
that require little training to administer to 
expensive and comprehensive instmments 


that require extensive training to administer 
and interpret. As a result, questions must be 
raised about the desired level of commitment 
needed to implement an AT outcome system. 
That is, will the profession need a cadre of 
assessment professionals (e.g., school 
psychologists) to administer, analyze, and 
interpret comprehensive evaluation tools? Or, 
will it need to focus on creating powerful 
assessment tools that are easily, validly, and 
reliably administered by a wide range of 
professionals? Hence, there is a need to 
understand the perspectives of the field 
regarding the expertise, training, and availability of 
the assessment instruments. 

Traditionally, assessment data has been 
collected through paper and pencil 
instruments. However, personal digital 
assistants (PDAs) and web-based data entry 
interfaces have emerged as potential tools for 
streamlining the time involved in data 
collection and improving the quality of data. 
While this vision is futuristic, is it practical 
given current levels of technology access and 
the availability of trained personnel? As a 
result, questions must be raised about the 
assumptions associated with data collection 
methods and preferences. That is, should an 
outcome system be built that only permits 
data to be uploaded from PDAs? Or, should 
users have to enter all data through a web- 
based interface? Hence, there is a need to 
understand the design features necessary to 
support legacy, as well as, emerging data 
collection techniques. 

In an attempt to improve the quality of AT 
outcome data, some have suggested the 
creation and validation of a select set of 
assessment instruments will resolve the issues 
associated with what data to collect. Others 
have argued that an outcome system must be 
inclusive in that the profession cannot dictate 
the specific data collection instmments that 
may be used in a given locale. As a result, 
questions must be raised about the desirability 
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of data assimilation tools built into an 
outcome system. That is, will the system 
accept data from a wide variety of assessment 
tools rather than a standardized list? To the 
extent possible, can the system help users 
understand the relationships among data 
collected using instruments that are based on 
a variety of assumptions and methodological 
approaches? Hence, there is a need to 
understand the desirability of data assimilation 
as a design principle for a prototype AT 
outcome measurement system. 

One of the intrinsic challenges associated with 
data interpretation involves seeing past the 
raw numbers in order to distill and understand 
patterns. This is particularly important in large 
data sets where the sheer volume of data can 
skew significant findings. As a result, 
questions must be raised concerning 
preferences for interacting with data. One 
promising application in this area involves 
visualization analysis tools. Hence, there is a 
need to understand the importance of 
designing tools that facilitate data reduction and 
visualisation as part of an outcome 
measurement system. 

One of the inherent difficulties associated 
with disability research is the intrinsic nature 
of making inter-individual comparisons. That 
is, the unique nature of an individual’s 
disability often precludes the opportunity to 
make comparisons with others. This is 
especially tme with low incidence disabilities 
where an individual may be the only one in a 
geographic area. Combine this challenge with 
the incidence of a specific type of AT and it is 
readily apparent that it is not possible to 
compare an individual’s performance to a 
group in order to understand the 
developmental context of enhanced 
performance. However, the ubiquitous nature 
of the Internet offers some intriguing 
possibilities for addressing these challenges. 
The ATOMS Project has outlined a concept 
we call, “dynamic norming.” Essentially this 


involves extracting data in a real-time database 
to make comparative norm groups. Users of 
the outcome system could make any number 
of comparisons using with simple search 
parameters to compares a client’s 
performance to (a) other individuals with 
similar disabilities who have used the same 
AT; (b) the types of services the client has 
received; or (c) a gap analysis of the 
compensation (AT) by comparing the 
performance results with the results of non¬ 
disabled individuals. Hence, there is a need to 
understand the perceived value of a dynamic 
norming component in an AT outcome system. 

Little is currently known about decision¬ 
making associated with AT outcome data. 
That is, if several professionals were to review 
the same data set, would they all come to the 
same conclusion about whether or not the AT 
devices and services were enhancing 
performance? The lack of attention to AT 
outcomes in personnel preparation suggests 
that there could be considerable variation in 
understanding and interpretation outcome 
data. As a result, there is an urgent need to 
understand the need for tools and resources 
that support data-based decision making. 



In order to obtain social validation data 
concerning the emerging design framework 
for a prototype assistive technology outcome 
system, data were collected as part of a 
presentation about the ATOMS Project at a 
large annual conference on AT (Edyburn & 
Smith, 2002). The conference attracts a 
diverse group of participants (e.g., special 
educators, occupational therapists, 
speech/language pathgologists, 

administrators, AT specialists, parents) that 
could be considered potential users of the 
proposed outcome measurement system. 
Approximately 80 participants attended the 
presentation and were invited to voluntarily 
complete an anonymous feedback form 
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during the presentation. A total of 58 
responses were received at the conclusion of 
the presentation. Clearly, the procedures 
represent a convenience sample which limit 
the generalizability of the results but provide 
valuable formative social validation evidence. 

Validation 

The seven components of the proposed AT 
outcome system were each communicated by 
the presenter through (a) verbal description, 
along with (b) a single PowerPoint slide to 
describe the function and possible utility of 
the component. Following the description of 
each component, session attendees were 
asked to validate the importance of the 
component by ranking on a five-point scale 
(1 —no value , 3 —some value , 5= great value) the 
perceived value of the strategy for including 
the proposed component in an AT outcome 
system. The anonymous questionnaires were 
returned to the presenter at the conclusion of 
the presentation. 

Analysis 

To analyze support for each component, data 
from the social validation exercise were 
counted and totaled. For this analysis, 
responses 4 and 5 were combined to indicate 
each respondent’s valuing of a component as 
being of significant value in an outcome system. 

Results 

The results of this social validation 
investigation are illustrated in Table 1. The 
respondents provided overwhelming support 
for the seven proposed components of an AT 
outcome system. That is, the following 
formative design principles were socially 
validated by a diverse group of potential users 
of a proposed outcome measurement system: 

1. A system should be designed using a 

theoretical framework supporting the 


relationship of variables involved in AT 
outcomes (e.g., satisfaction, performance, 
use, quality of life, etc.). 95% of the 
respondents indicated that this 
component would be of significant value. 

2. Outcome assessment instruments 
should not require extensive training and 
expertise to administer and should be 
readily available. 86% of the respondents 
indicated that this component would be of 
significant value. 

3. Data collection tools should support 
traditional paper and pencil instruments as 
well as portable handheld devices (PDAs) 
and web-based interfaces. 100% of the 
respondents indicated that this 
component would be of significant value. 

4. Tools should be available for 
assimilating data from multiple sources and 
instruments in ways that allow 
comparisons to be readily made. 83% of 
the respondents indicated that this 
component would be of significant value. 

5. Easy to use tools must be provided to 
allow professionals and end users to 
reduce multiple scores into easy-to- 
understand visuals that foster interpretation of 
the data. 86% of the respondents indicated 
that this component would be of significant 
value. 

6. Given the unique and low incidence 
nature of many AT interventions, tools 
should be available that facilitate dynamic 
norming (individual and group 
comparisons) of the AT outcomes data. 
That is, it should allow comparisons of an 
individual’s scores with others like 
him/her in terms of their disability, length 
of device use, type of device, 
environment, expectations, and other 
variables? 88% of the respondents 
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indicated that this component would be of 

significant value. 

7. Tools should be available to facilitate 
decision-making based on the data. 78% of 
the respondents indicated that this 
component would be of significant value. 
This item received the lowest rating of the 
seven components. This may be a 
reflection of the thinking of several 
respondents who questioned the value the 
entire system if it did not address this 
component. 

Discussion 

The discipline of AT outcomes is still in its 
infancy and suffers from a lack of data to 
support claims about the effectiveness of AT. 
As a result, there is a need for considerable 
conceptual work to guide research on AT 
outcomes (Lenker & Paquet, 2003). 

The ATOMS Project has proposed the 
development of an AT outcomes information 
system that would facilitate the collection and 


preliminary study reveal a high level of 
support for seven components of a proposed 
system. 

The current study is subject to a number of 
limitations primarily due to the conceptual 
format of the proposed outcome system 
prototype and lack of a working prototype. 
Also, the use of a convenience sample limits 
the application of the results. Despite these 
shortcomings, the social validation process 
provides important formative evaluation of 
the current development efforts and engages 
the profession in a dialogue about a shared 
vision concerning the purpose and use of an 
AT outcome system. Obviously, additional 
research is needed concerning the 
development and use of AT outcome 
measurement system. 

Outcomes and Benefits 

Preliminary research and development work 
by the ATOMS Project reveals the following 
insights associated with the outcomes and 


TABLE 1 

Percentage of Respondents Valuing Proposed Components of an AT Outcome System 


Component 


Perceived Value of This Strategy (%) 



No Value 

Some Value 

Great Value 

Theoretical Framework 

0 

4 

96 

Expertise, Training & 
Availability 

0 

9 

91 

Data Collection 

0 

2 

98 

Data Assimilation 

2 

14 

84 

Data Reduction & 
Visualization 

2 

12 

86 

Making Data Meaningful 

3 

9 

88 

Applications of Data 
(Decision-Making) 

5 

13 

82 


use of outcome data. The results of this 
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benefits of AT: 

1. The suggestion that AT outcomes 
involves more than simple consumer 
satisfaction received an encouraging 
reception by the participants in this study. 
In response, they supported efforts to 
develop conceptual models that will 
enable the profession to develop data- 
based evidence about AT outcomes. 

2. Participants in this investigation 
preferred data collection instruments that 
do not require extensive training to 
implement and are inclusive of a variety of 
assessment instruments and data 
collection tools (e.g., paper and pencil, 
PDA, web-based interfaces) rather than 
approaches that involve extensive clinical 
data collection efforts and expertise to 
administer. This work also supports the 
desirability of initiatives within the AT 
industry to build data capture mechanism 
into AT devices. 

3. Potential users of AT outcome data 
expressed a preference for tools that help 
them understand the meaning of the 
outcome data they have collected. This, in 
turn, would facilitate appropriate decision¬ 
making. 

4. The concept of dynamic norming, 
extracting data in the database to make 
comparative norm groups, was positively 
received by the participants in this study. 
Social validation of this unique design 
principle in developing AT outcome 
systems is important given that it 
represents an advance that would be 
impossible to achieve with traditional 
approaches to tests and measurement. 

Conclusion 

The knowledge base concerning how to 

measure the outcomes of AT is still in its 


infancy. Considerable work is needed to 
define the theoretical constructs necessary to 
create data collection systems that will 
produce outcome data for subsequent analysis 
and understanding of the impact AT. 

This report presents a brief summary and 
analysis of some initial efforts of the ATOMS 
project to define potential components of an 
AT outcome system. Future studies will 
explore additional design considerations, 
usability, and applications of an AT outcome 
system. For additional information, please 
contact: atoms@uwm.edu. 


This work was supported in part by Grant # 
H133A010403 from the National Institute on 
Disability and Rehabilitation Research, U.S. 
Department of Education to the University of 
Wisconsin-Milwaukee. Points of view or 
opinions stated in this article do not 
necessarily represent official agency positions. 
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